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Abstract 

The purpose of this report is to explore the content-related validity evidence supporting the 
English language arts (ELA) components of the ETS® National Observational Teaching Exam 
(NOTE) assessment series, a kindergarten through 6th-grade teacher licensure assessment. To 
establish the content knowledge required for the effective teaching of ELA in elementary school, 
we (a) identified content knowledge categories through the use of an expert panel and (b) 
surveyed a sample of 279 educators to verify that this body of content knowledge is indeed 
necessary and reasonable for the effective practice of beginning elementary school teachers 
teaching ELA. We report information regarding the importance and relevance of ELA content 
knowledge areas for both elementary school teachers and faculty members who prepare 
elementary school teachers. Implications of this work for the ELA components of the NOTE 
assessment series are discussed. 

Key words: English language arts, content validity, teacher licensure, elementary school 
teaching, content knowledge for teaching 
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The purpose of this report is to explore the content-related validity evidence supporting 
the English language arts (ELA) content knowledge for teaching (CKT) components of the ETS® 
National Observational Teaching Exam (NOTE) assessment series. NOTE is a kindergarten 
through 6th-grade licensure assessment developed in a collaboration between Educational 
Testing Service (ETS) and TeachingWorks (http://www.teachingworks.org). The NOTE 
assessment series is designed to measure a prospective elementary school teacher’s ability to 
translate his or her knowledge of content and of teaching into effective teaching practice. The 
NOTE assessment series includes two components. One component includes standardized 
performance assessments that focus on three high-leverage practices (HLPs) for teaching: 
modeling and explaining content, evaluating student thinking, and leading a classroom 
discussion. TeachingWorks defines HLPs as “the basic fundamentals of teaching. These 
practices are used constantly and are critical to helping students learn important content. These 
high-leverage practices are used across subject areas, grade levels, and contexts” (2016b, para. 

2). The NOTE assessment series assesses high-leverage content, defined as the specific topics, 
practices, and texts that have been put forward by TeachingWorks as foundational to the K-12 
curriculum and crucial for beginning teachers to be able to teach (TeachingWorks, 2016a). This 
content, organized by subject area and grade level, is rooted in national and state standards for 
student learning that have been crafted with the involvement of key professional groups. 

The second component focuses on CKT. CKT is a theory derived via job analysis and 
based in practice that outlines the content knowledge required for teaching a subject (ETS, 

2011). Each component must include tasks that identify CKT necessary for the effective teaching 
of ELA. According to the Standards for Educational and Psychological Testing (American 
Educational Research Association [AERA], American Psychological Association [APA], 
National Council on Measurement in Education [NCME], 2014), a critical element of licensure 
assessments that focus on beginning teaching proficiency in subjects such as ELA is valid 
frameworks that define the ELA CKT domains. 

The remainder of this report is organized as follows. The first section provides a brief 
overview of the role of content-related validity evidence in licensure assessment. The second 
section describes the process we used to generate the ELA CKT framework. The third section 
describes the study design, methods, and results. Finally, we conclude by discussing the 
implications of this work for the ELA components of the NOTE assessment. 
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Content-Related Validity Evidence 

A chief function of licensure assessments is to differentiate between candidates who 
possess the knowledge and skills required for beginning practice and those who do not (Clauser, 
Margolis, & Case, 2006; Smith & Hambleton, 1990). Passing scores on licensure assessments 
indicate that candidates possess the necessary abilities to teach effectively and in a way that 
ensures public welfare (AERA, APA, & NCME, 2014; Raymond & Luecht, 2013). 

Licensure assessments often measure the knowledge, skills, and abilities (KSAs) required 
for performing elements of a job rather than performance on actual job tasks (Wang, Schnipke, 

& Witt, 2005). Test specifications can describe assessment content and the KSAs that should be 
measured by the assessment (Raymond, 1996) as well as provide a critical foundation for 
validity evidence (Ebel & Frisbie, 1991). Consequently, it is essential to pinpoint the KSAs 
necessary for performing job tasks to design test specifications that are clearly related to 
performance in a given profession. 

One job-analytic strategy that is often used to define the content domain for a licensure 
assessment involves having a panel of subject matter experts develop a compilation of KSAs 
linked to the effective execution of job tasks (Rosenfeld & Tannenbaum, 1991; Tannenbaum & 
Wesley, 1993; Wang et al., 2005). Surveys of a large sample of qualified practitioners in a given 
profession are commonly used to then verify the judgments about KSAs made by a panel of 
subject matter experts (Rosenfeld & Tannenbaum, 1991; Tannenbaum & Wesley, 1993). This 
sample of experts is often asked to rate elements of the content domain regarding their relevance 
and importance (Kane, Kingsbury, Colton, & Estes, 1989; Raymond, 2005; Tannenbaum & 
Wesley, 1993) so that this information can contribute to the development of empirically derived 
specifications for a licensure assessment. 

The process of generating KSAs involves consulting with a diverse set of subject matter 
experts that span a variety of work backgrounds and job positions (Raymond, 2001; Raymond & 
Luecht, 2013). Attributes such as race, ethnicity, gender, urban/rural setting, and geographic 
location should be considered when selecting qualified subject matter experts for this purpose 
(Clauser et al., 2006; Tannenbaum & Wesley, 1993), as it is critical that the experts provide 
perspectives from a diverse array of demographic groups. 

It is crucial that all pertinent subgroups of experts agree on the KSAs that are relevant 
and important for performing the job (Tannenbaum & Rosenfeld, 1994). Moreover, it is possible 
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that the group of experts as a whole may indicate that a KSA is not especially relevant or 
important, suggesting that the KSA in question may not be suitable for a licensure assessment. 
However, group comparisons may show that the same KSA is perceived as being both relevant 
and important by most of those in a particular subgroup or geographic region, indicating that the 
KSA should indeed be measured by the licensure assessment (Raymond, 2005). Similarly, 
different constituencies of experts, such as practicing teachers and faculty members at teacher 
preparation programs, may differ in their judgments of the relevance and importance of 
particular KSAs. 

To establish the CKT for ELA in elementary school, we used this process by first 
establishing CKT categories through the use of an expert panel and then surveying a larger 
sample of educators to verify that this body of content knowledge is indeed necessary and 
reasonable for the effective practice of beginning elementary school teachers teaching ELA. 

Establishing a CKT Framework for ELA 

As part of the development process for NOTE, an advisory panel of 10 literacy experts 
proposed a framework that defines the CKT needed for effective beginning teaching in 
elementary ELA. The advisory panel included leading literacy scholars representing expertise in 
early literacy, writing, comprehension, multicultural issues in ELA, English language learners, 
literacy in content areas, student assessment, accreditation, and teacher education. 1 The advisory 
panel recognized that part of their charge was to winnow the huge range and depth of knowledge 
required for teaching ELA to propose the most critical content that both is needed for initial entry 
into the profession and could serve as the basis of a framework for a licensure exam. 

The development process was guided by the current research and best practices about the 
defining characteristics of ELA CKT. As an academic subject, the CKT for ELA is best 
described as a set of related practices rather than a body of knowledge to be acquired. Reading, 
writing, speaking, and listening are not so much subjects that one learns about as they are tools 
that one learns how to use in the acquisition of specific knowledge, usually knowledge that is 
viewed as the province of another domain, such as science, history, or literature. All CKT 
domains entail elements of practice; for example, science and history each possesses its own 
rules of evidence in making arguments. However, ELA stands out from other elementary school 
subjects as one in which processes and practices dominate. First, a set of practices, usually taught 
in grades K-3, defines what the Common Core State Standards (CCSS) document (National 
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Governors Association Center for Best Practices & Council of Chief State School Officers [NGA 
Center & CCSSO], 2010) refers to as foundational skills. These are the fundamental discrete 
skills that allow students to deal with the cipher — the code that allows students to decode written 
language into spoken language (e.g., the phonemic awareness and phonics skills that lead to 
accurate, fluent oral reading) or encode oral language into its written form (spelling, grammar, 
usage, and conventions of written language). Beyond these foundational skills, ELA is more 
about applying and enriching a set of literacy tools across genres, disciplines, and 
contexts (Goldman et al., 2016; NGA Center & CCSSO, 2010; Pearson, Moje, & Greenleaf, 
2010). These tools include not only the basic skills to read, comprehend, and compose written 
text, but also general literacy skills in areas such as reasoning, exposition, narration, 
argumentation, and so forth (Goldman et al., 2016). Further, competence in literacy also includes 
nurturing critical dispositions and the habits in reading, writing, speaking, and listening (e.g., 
stamina, risk taking, regular practice) that drive student effort and motivation (CCSS, 2010). 
Thus, ELA CKT includes, and arguably is defined by, literacy practices and their associated 
language tools (National Research Council, 2012). 

The ELA advisory panel recognized that the content that students are expected to learn is 
the primary basis for identifying the ELA CKT that defines the work of teaching. However, 
student content and CKT are not isomorphic. Many professional aspects of ELA CKT bear 
strong consequences for what and how students are taught (Snow, Griffin, & Bums, 2005; 
Wong-Fillmore & Snow, 2005). ELA CKT includes many types of content-related tasks and 
knowledge that extend beyond the student curriculum, such as understanding how literacy skills 
are leveraged across content areas; the developmental progressions of content across grade 
levels; the variation in student development; and how racial, cultural, and language background 
shape literacy practice (Pearson, Griffo, & Phelps, 2015). Teachers may use this elaborated 
knowledge to inform student evaluation and support. For example, teacher knowledge of dialect 
interference between an English vernacular and academic English helps inform whether to 
attribute language errors to language interference, student misconceptions, or developmental 
delays (Godley, Sweetland, & Wheeler, 2006). Thus, teachers must possess broader and deeper 
ELA CKT that frames what components are most important, how they are interrelated, and a 
trajectory of instruction. 
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Attention to both ELA CKT and the additional professional knowledge about ELA that is 
used in teaching this content guided the development of an ELA CKT framework for the NOTE 
assessment. The CKT development was organized into three phases (Pearson et al., 2015). In 
Phase 1, a subgroup of the full ELA advisory committee considered a variety of ways to organize 
literacy content for teaching. The goal of this stage of work was to lay out a reasonable set of 
viable possibilities and consider the advantages and disadvantages of different taxonomies. After 
reviewing a number of organizations, the panel decided to focus on categories that aligned with 
widely used student standards with attention to both the content students are expected to master 
and the content that defines the work of teaching that student content. This stage of development 
produced a preliminary set of eight CKT categories based on the committee’s professional 
knowledge and a review of pertinent literature in the field: word analysis, fluency, vocabulary, 
reading comprehension, listening and speaking, oral and written language conventions, writing 
strategies and applications, and genre analysis. 

Each of these areas was then evaluated to ensure that the CKT strand met the criteria 
developed by TeachingWorks (2016a) to identify content that is foundational to the K-12 
curriculum and critical for beginning teachers to be able to teach skillfully. Criteria identified 
CKT that meet the following conditions: (a) foundational to the ideas and skills of the K-12 
curriculum, (b) taught in some form or another across several K-6 grade levels, (c) occupies 
substantial space in the curriculum, (d) fundamental to students’ learning and is often a site for 
students’ difficulties when it is not well taught and learned, and (e) often known only 
superficially by prospective teachers or is new to them (i.e., content that they may have never 
had the opportunity to learn). The advisory panel identified two additional criteria specific to 
ELA: highly transferable across ELA modalities and other disciplines and highly transferable 
from school to nonschool settings. 

In Phase 2 of the development, these categories were further refined to bring them into 
close alignment with the CKT categories widely used in student standards. To elaborate these 
categories and refine the language used in the accompanying descriptions, subteams of the ELA 
committee conducted an extensive review of the literature and existing ELA-related frameworks 
(e.g., CCSS [NGA Center & CCSSO, 2010], InTASC [CCSSO, 2011], International Reading 
Association/National Council of Teachers of English [IRA/NCTE, 1996] standards). The 
subgroups followed a recursive process of research and review, drafting, discussing, and revising 
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potential frameworks based on feedback from members of the ELA committee, validity 
researchers at ETS, and members of the ETS development team. 

In Phase 3, the CKT standards from Phase 2 were revised based on concerns raised by the 
full ELA committee. The committee suggested that the focus on student standards that had 
guided the initial development had overlooked important content aspects of student learning such 
as literacy acquisition, student literacy dispositions, and language variation. Also, the focus on 
student content had failed to adequately represent CKT specific aspects of professional practice 
and knowledge. Additional categories were suggested by the full committee including, for 
example, interactions among the literacy modalities of reading, writing, speaking, and listening; 
reading processes and literacy development; children’s literature; text difficulties and readability; 
and technology integration. To ensure that the CKT areas were comprehensive, another subteam 
was formed to systematically review widely used professional standards for teaching ELA (e.g., 
the International Reading Association [2010] and the International Dyslexia Association [IDA, 
2010]). This review led to an expanded set of 24 CKT areas. Through subsequent review and 
revision, the full ELA committee revised and expanded the descriptors for each category to 
ensure that the full range of ELA CKT was referenced in the content knowledge framework. 
Descriptions of the 24 CKT areas are included in the appendix. 

We surveyed educators in the field of ELA to establish the importance of the 24 CKT 
areas for beginning elementary school teachers identified by our panel of experts. The study 
draws on a sample of practicing teachers and teacher educators to collect evidence of 
endorsement for the CKT categories. Results are presented for overall endorsement and by 
relevant subgroups including professional role, race, professional experience, and grade level 
emphasis. 


Method 

Sample 

A mailing list of 8,841 educators was sampled from a large, national database of teachers 
and teacher preparation faculty provided by an educational marketing firm. The sample included 
elementary school teachers and college faculty who prepare elementary school teachers. An 
approximately equal number of teachers were sampled from each of the four United States 
Census regions. Black or African American and Hispanic/Latino teachers and faculty from 
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minority-serving institutions were oversampled to increase the likelihood of having sufficient 
sample sizes to conduct group analyses. 

Teachers were assigned to one of two versions of a survey (one for mathematics and one 
for ELA) depending on their current teaching assignments, as this study was part of larger 
investigation. Teachers who only taught mathematics were assigned to the mathematics version, 
and teachers who only taught ELA were assigned to the ELA version. Teachers who indicated 
they taught both ELA and mathematics (more than 75% of the sample) or neither (approximately 
4% of the sample) were randomly assigned to one of the two versions. Faculty also were 
randomly assigned to one of the two versions. 

Of the original 8,841 educators contacted, 700 emails were not deliverable. Therefore, the 
number of educators successfully contacted was 8,141. Of these, 607 (or 7.5%) completed one of 
the two versions of the survey. An additional 31 educators were forwarded the survey by 
colleagues and completed it. In total, 638 educators completed either the mathematics or ELA 
version of the online survey. Of the respondents, 387 (or 61%) indicated they were elementary 
school teachers and 202 (or 32%) indicated they were college faculty. The remaining 49 
respondents (or 8%) indicated they were administrators, held other education-related positions, 
or preferred not to provide information regarding their current position. Given the purpose of the 
survey, the 49 respondents who did not indicate they were teachers or faculty were removed 
from the sample. 

The resulting sample — currently licensed teachers and college faculty currently preparing 
elementary school teacher candidates — includes 569 respondents, comprising 385 teachers and 
184 college faculty across the two versions (mathematics and ELA) of the survey. The ELA 
subsample for the following analyses includes currently licensed teachers and college faculty 
currently preparing elementary school teacher candidates ( n = 279). Of the respondents, 197 (or 
71%) indicated they were teachers (elementary school) and 82 (or 29%) indicated they were 
college faculty. 

While the overall response rate 2 for the survey was 7.5%, the sample of elementary 
school teachers reasonably reflects the composition of the national population when compared to 
the National Center for Education Statistics (NCES) 2011-12 School and Staffing Survey 
(SASS), taking into account the sampling design considerations mentioned above. The sample of 
teachers slightly overrepresents the percentage of Black teachers (10% in the sample compared 
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to 7.1% nationally) and Hispanic teachers (10% in the sample compared to 8.7% nationally) 
compared to the latest SASS results (Goldring, Gray, & Bitterman, 2013). The sample 
approximately mirrors elementary school teachers nationally in terms of gender, with 
approximate 90% of elementary school teachers being female nationally, compared to 85% in 
this sample (Goldring et al., 2013). Table 1 provides a summary of background information for 
the ELA teachers and faculty comprising the sample. 

Administration Procedure 

An online survey was used to contact educators via email and letter, to invite them to 
complete the survey. Participants were paid $25 via a gift card in exchange for their participation 
in the study, which took approximately 45 minutes to complete. Three email follow-up 
reminders were sent during the data collection period. 

Survey Instrument 

Participants were asked to provide relevance and importance judgments for each of the 
24 ELA CKT areas. For both relevance and importance judgments, participants were asked to 
consider the importance for a beginning teacher’s ability to effectively teach the subject. For 
each of the 24 ELA CKT areas, the two content-related validity questions posed to educators 
were as follows: 

1. Is knowing how to teach this content area relevant to a beginning elementary school 
teacher’s ability to be a safe and effective ELA teacher? 

2. If knowing how to teach this content area is relevant, how important is it to a 
beginning elementary school teacher’s ability to be a safe and effective ELA teacher? 

If an educator indicated a CKT area was relevant, the educator then rated the importance 
of the CKT area using a 6-point judgment scale ranging from 1 ( not at all important ) to 6 
(i extremely important). Therefore, importance ratings were only collected from respondents who 
judged the CKT as relevant. Judgments about relevance were dichotomous, such that a CKT was 
deemed either relevant or irrelevant. 
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Table 1. Background Information — Overall Sample (N = 279) 


Item 

Teachers (N = 197) 

Faculty ( N = 82) 

Gender 

Female 

174 (88%) 

63 (77%) 

Male 

16 (8%) 

18 (22%) 

Missing/prefer not to answer 

7 (4%) 

1 (1%) 

Race/ethnicity 

American Indian or Alaska Native 

2 (1%) 

0 (0%) 

Asian or Asian American 

0 (0%) 

3 (4%) 

Black or African American 

21 (11%) 

8 (10%) 

Hispanic/Latino 

24(12%) 

3 (4%) 

Native Hawaiian or Other Pacific Islander 

1 (1%) 

0 (0%) 

White 

135 (69%) 

64 (78%) 

Two or more races 

2 (1%) 

2 (2%) 

Other/prefer not to answer/missing 

12 (6%) 

2 (2%) 

Geographic region 

Northeast 

37 (19%) 

14 (17%) 

Midwest 

55 (28%) 

22 (27%) 

South 

63 (32%) 

31 (38%) 

West 

42 (21%) 

15 (18%) 

Current teaching assignment 2 

Lower (Grades K-3) 

103 (52%) 

_ 

Upper (Grades 4-6) 

75 (38%) 

— 

Years of experience 

3 years or less 

10 (6%) 

6 (7%) 

4 to 9 years 

40 (20%) 

24 (29%) 

10 to 14 years 

44 (22%) 

24 (29%) 

15 years or more 

102 (52%) 

28 (35%) 

Missing 

1 (<1%) 


Mentored or supervised student teachers 

Yes 

96 (49%) 

49 (60%) 

No 

99 (50%) 

32 (39%) 

Missing 

2 (1%) 

1 (1%) 

Type of school 

Public (noncharter) 

173 (88%) 


Public (charter) 

12 (6%) 

— 

Private 

11 (6%) 

— 

Missing 

1 (<1%) 

— 

School or institution location 

Urban 

67 (34%) 

26 (32%) 

Suburban 

82 (42%) 

31 (38%) 

Rural 

48 (24%) 

25 (30%) 

Minority-serving institution 

Yes 


25 (30%) 

No 

— 

53 (65%) 

Designation not available 

— 

4 (5%) 


“The number of teachers teaching at the lower and upper elementary grades does not sum to the number of 
elementary school teachers, as some teachers taught across the lower (Grades K-3) and upper (Grades 4-6) 
elementary grades or taught prekindergarten. 
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After completing the relevance and importance ratings for the 24 CKT categories, 
participants were asked to select the five most and five least critical CKT items from the list. 
This rating procedure provided insight into which smaller set of CKT items are viewed as most 
and least critical for initial practice. 


Analysis 

The analyses in this report were focused in two main categories. The first category was to 
describe patterns in average relevance and importance judgment ratings in various ways, starting 
with all participants, and then make comparisons between teachers and faculty, lower elementary 
and upper elementary teachers, teachers across race/ethnicity groups, and teachers across 
geographic regions. Comparisons were made using effect sizes (Cohen, 1988) where the mean 
difference between two groups was divided by a combination of group sample sizes and standard 
deviations. In the case of race/ethnicity, White teachers were used as the reference group and in 
the case of geographic region, Northeastern teachers were used as the reference group. The 
second category of analyses was to index agreement between relevance and importance using 
intraclass correlations (ICC[2]; Shrout & Fleiss, 1979). 

Results 

Overall Agreement Concerning Relevance and Importance Judgments 

Across the 24 ELA CKT categories, the majority of educators, more than three quarters, 
agreed that the CKT areas are relevant for effective practice for beginning elementary school 
teachers teaching ELA. The ICC[2] (Shrout & Fleiss, 1979) indexing agreement among 
educators regarding their relevance ratings across the 24 CKT areas is .92 (95% Cl [.86, .96]). 

The importance judgments, averaged across both teachers and faculty, exceeded the 
threshold of 4.2 recommended by Tannenbaum and Rosenfeld (1994) for each of the 24 ELA 
CKT categories. 3 Because respondents only made ratings for importance if they indicated a CKT 
area was relevant, this limitation resulted in some missing cases. However, given the range in 
average importance judgments and associated levels of variation, there was no substantive 
evidence of floor or ceiling effects in the data. To compute intraclass correlations, we made the 
assumption that if a CKT area was not considered relevant, it would also be considered not at all 
important. Therefore, we imputed a value of 1 for missing cases. The ICC[2] (Shrout & Fleiss, 
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1979) indexing agreement among educators regarding their importance ratings across the 24 
CKT areas is .98 (95% Cl [.96, .99]). 

Group Differences in Relevance and Importance Ratings 

The percentages of educators indicating each CKT area was relevant were above 90% for 
both teachers and faculty for 23 of the 24 CKT categories. One CKT area, sources of language 
variability (CKT 17), was judged relevant by 76% of teachers and 91% of faculty. Relevance 
judgments for this one CKT area were the lowest, and it was the only area where teachers and 
faculty differed by more than 10 percentage points. 

Table 2 summarizes educators’ judgments regarding the importance of each of the 24 
CKT categories. Results are presented for teachers, faculty, and the total sample. The absolute 
value of the differences in average importance judgments between teachers and faculty was 
lower than 0.25 (on the 6-point scale) for 18 of the 24 categories. Effect sizes (ES) were lower 
than 0.20 (Cohen, 1988) for 15 of the 24 categories. Teachers and faculty differed most on CKT 
4: Phonics and Word Recognition (diff. = 0.31; ES = 0.41); CKT 15: Integration and Application 
of Knowledge (diff. = 0.29; ES = 0.33); CKT 17: Sources of Language Variability (diff. = 0.26; 
ES = 0.27); CKT 20: Role of Engagement in Reading and Writing (diff. = 0.28; ES = 0.31); CKT 
22: Development of Word Reading (diff. = 0.25; ES = 0.28); and CKT 24: Development of Early 
Oral Language (diff. = 0.28; ES = 0.29). Teachers, on average, judged CKT 4: Phonics and 
Word Recognition and CKT 22: Development of Word Reading more important than faculty did; 
faculty judged the remaining four areas (CKT 15, CKT 17, CKT 20, and CKT 24) to be more 
important. 
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Table 2. Summary of Importance Judgments for Content Knowledge for Teachers (CKT) 
Areas for Teachers, Faculty, and Overall 


CKT area 

Teachers 

Faculty 

Overall 

1 

5.28 (0.86) 

5.16(0.78) 

5.24 (0.84) 

2 

5.53 (0.74) 

5.30 (0.77) 

5.46 (0.75) 

3 

5.43 (0.81) 

5.20 (0.84) 

5.36 (0.82) 

4 

5.57 (0.70) 

5.26 (0.85) 

5.48 (0.76) 

5 

5.23 (0.82) 

5.28 (0.69) 

5.24 (0.78) 

6 

5.11 (0.79) 

5.18 (0.74) 

5.13 (0.77) 

7 

5.26 (0.77) 

5.33 (0.76) 

5.28 (0.77) 

8 

4.79 (0.86) 

4.74 (0.85) 

4.77 (0.85) 

9 

4.81 (0.86) 

4.75 (0.86) 

4.79 (0.86) 

10 

4.83 (0.88) 

4.75 (0.81) 

4.81 (0.86) 

11 

4.99 (0.86) 

5.10(0.88) 

5.03 (0.87) 

12 

5.27 (0.74) 

5.09 (0.82) 

5.22 (0.77) 

13 

4.84 (0.85) 

4.96 (0.81) 

4.88 (0.84) 

14 

5.12 (0.89) 

5.20 (0.80) 

5.14(0.86) 

15 

4.69 (0.89) 

4.99 (0.90) 

4.79 (0.90) 

16 

4.71 (0.90) 

4.77 (0.90) 

4.73 (0.90) 

17 

4.30 (0.93) 

4.56(1.00) 

4.39 (0.96) 

18 

5.13 (0.90) 

5.25 (0.78) 

5.17(0.86) 

19 

4.68 (0.89) 

4.84 (0.95) 

4.72 (0.91) 

20 

4.90 (0.90) 

5.18 (0.87) 

4.98 (0.90) 

21 

5.05 (0.95) 

4.97(1.04) 

5.03 (0.98) 

22 

5.04 (0.84) 

4.79 (0.99) 

4.97 (0.89) 

23 

4.67 (0.95) 

4.51 (0.93) 

4.62 (0.94) 

24 

5.07(1.02) 

5.35 (0.88) 

5.15 (0.99) 

Minimum 

4.30 

4.51 

4.39 

Maximum 

5.57 

5.35 

5.48 

Sample size 

149-194 

75-82 

224-275 


Note. The importance scale is a 6-point scale: 1 ( not at all important ), 2 (of little importance), 

3 (of some importance), 4 (moderately important), 5 (very important), 6 (extremely important)', 
respondents who judged the practice not relevant are not included in the calculation of the 
average importance judgment. The 24 CKT areas are summarized in the appendix. 

In addition to considering the importance judgments of teachers overall, average 
judgments for teachers who are currently teaching lower (kindergarten through Grade 3) and 
upper (Grades 4 through 6) elementary grades were examined. Table 3 summarizes teachers’ 
judgments, disaggregated by current grade levels taught. Respondents to the survey were 
instructed to consider the full range of elementary grades when making their judgments; 
disaggregating teachers by current teaching assignment revealed small differences, less than 
0.25, in importance judgments by current experiences for 20 of the 24 CKT categories. The 
remaining four CKT categories — CKT 3: Phonological Awareness; CKT 22: Development of 
Word Reading; CKT 23: Development of Word Spelling; and CKT 24: Development of Early 
Oral Language — had importance judgment differences greater than 0.25, with all being higher 
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for teachers currently teaching at the lower grades. Effect sizes less than 0.20 were present for 16 
of the 24 CKT categories, but for the four listed here, effect sizes were 0.37 or higher. 


Table 3. Summary of Importance Judgments for Content Knowledge for Teachers (CKT) 
Areas by Current Grade Level Taught 


CKT area 

Lower (K-3) 

Upper (4-6) 

Difference 

1 

5.30 (0.85) 

5.23 (0.89) 

0.06 (0.07) 

2 

5.58 (0.64) 

5.41 (0.90) 

0.18 (0.23) 

3 

5.54 (0.69) 

5.25 (0.95) 

0.30 (0.37) 

4 

5.63 (0.64) 

5.51 (0.75) 

0.12(0.18) 

5 

5.28 (0.80) 

5.16(0.86) 

0.12(0.15) 

6 

5.10(0.76) 

5.08 (0.87) 

0.02 (0.02) 

7 

5.20 (0.78) 

5.35 (0.77) 

0.15 (0.20) 

8 

4.85 (0.86) 

4.67 (0.88) 

0.19(0.21) 

9 

4.82 (0.78) 

4.76 (0.95) 

0.06 (0.07) 

10 

4.88 (0.88) 

4.77 (0.89) 

0.11 (0.12) 

11 

5.04 (0.80) 

4.90 (0.95) 

0.14(0.16) 

12 

5.25 (0.73) 

5.32 (0.76) 

0.07 (0.10) 

13 

4.82 (0.79) 

4.83 (0.96) 

0.01 (0.01) 

14 

5.13 (0.84) 

5.08 (0.90) 

0.05 (0.05) 

15 

4.73 (0.79) 

4.66 (0.96) 

0.07 (0.08) 

16 

4.68 (0.81) 

4.73 (0.98) 

0.05 (0.06) 

17 

4.28 (0.86) 

4.24(1.04) 

0.04 (0.04) 

18 

5.19(0.86) 

5.08 (0.92) 

0.10(0.12) 

19 

4.62 (0.84) 

4.68 (0.97) 

0.06 (0.07) 

20 

4.94 (0.87) 

4.88 (0.93) 

0.06 (0.07) 

21 

5.12(0.91) 

4.90(1.05) 

0.22 (0.23) 

22 

5.16(0.77) 

4.81 (0.91) 

0.35 (0.42) 

23 

4.88 (0.82) 

4.42(1.05) 

0.46 (0.50) 

24 

5.22 (0.86) 

4.80(1.15) 

0.42 (0.43) 

Minimum 

4.28 

4.24 

0.01 

Maximum 

5.63 

5.51 

0.46 

Sample size 

83-102 

50-75 



Note. The importance scale is a 6-point scale: 1 ( not at all important ), 2 (of little importance ), 

3 (of some importance), 4 (moderately important ), 5 (very important ), 6 (extremely 
important)', respondents who judged the practice not relevant are not included in the 
calculation of the average importance judgment. The 24 CKT areas are summarized in the 
appendix. 

Importance judgments broken down by educator ethnicity are shown in Table 4. 
Importance judgments by Hispanic respondents differed most from White respondents on CKT 
8: Forms and Functions of Fanguage; CKT 10: Text Types; CKT 15: Integration and Application 
of Knowledge; CKT 17: Sources of Fanguage Variability; CKT 20: Role of Engagement in 
Reading and Writing; and CKT 22: Development of Word Reading; differences on these areas 
ranged from 0.37 to 0.58 on a 6-point scale with effect sizes ranging from 0.42 to 0.70. Two of 
these importance judgments by Black respondents also differed most from White respondents 
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(CKT 10 and CKT 22) ranging from 0.25 to 0.42 on a 6-point scale with effect sizes of 0.27 and 
0.47, respectively. 


Table 4. Summary of Importance Judgments for Content Knowledge for Teachers (CKT) 
Areas by Race/Ethnicity 


CKT area 

African American 

Hispanic/Latino 

White 

Overall 

1 

5.14(1.01) 

4.96 (0.88) 

5.32 (0.86) 

5.25 (0.88) 

2 

5.60 (0.60) 

5.38 (0.77) 

5.57 (0.74) 

5.54 (0.73) 

3 

5.30 (0.73) 

5.58 (0.65) 

5.42 (0.85) 

5.43 (0.81) 

4 

5.45 (0.69) 

5.63 (0.65) 

5.58 (0.72) 

5.57 (0.70) 

5 

5.35 (0.88) 

5.38 (0.71) 

5.17(0.83) 

5.22 (0.82) 

6 

5.05 (0.83) 

5.30 (0.70) 

5.06 (0.80) 

5.09 (0.79) 

7 

5.10(0.91) 

5.50 (0.66) 

5.22 (0.78) 

5.24 (0.79) 

8 

4.68 (0.89) 

5.27 (0.70) 

4.69 (0.86) 

4.76 (0.86) 

9 

4.94 (0.87) 

5.09(1.00) 

4.76 (0.83) 

4.83 (0.86) 

10 

4.55 (0.89) 

5.21 (0.72) 

4.80 (0.91) 

4.82 (0.89) 

11 

5.00(1.11) 

5.13 (0.81) 

4.95 (0.86) 

4.98 (0.88) 

12 

5.25 (0.85) 

5.50 (0.59) 

5.22 (0.77) 

5.26 (0.76) 

13 

4.84(1.01) 

5.17(0.58) 

4.79 (0.86) 

4.85 (0.85) 

14 

5.05 (1.02) 

5.26 (0.69) 

5.08 (0.92) 

5.10(0.91) 

15 

4.61 (1.24) 

5.10(0.70) 

4.61 (0.88) 

4.68 (0.91) 

16 

4.65 (1.04) 

5.00 (0.87) 

4.67 (0.90) 

4.71 (0.91) 

17 

4.21 (0.89) 

4.80 (0.70) 

4.22 (0.97) 

4.31 (0.94) 

18 

5.05 (0.92) 

5.17(0.78) 

5.17 (0.92) 

5.16(0.90) 

19 

4.50 (0.95) 

4.83 (0.94) 

4.70 (0.88) 

4.70 (0.90) 

20 

5.06(1.11) 

5.26 (0.62) 

4.80 (0.92) 

4.89 (0.92) 

21 

5.00 (0.88) 

5.35 (0.78) 

5.05 (0.99) 

5.08 (0.95) 

22 

5.37 (0.76) 

5.32 (0.72) 

4.95 (0.89) 

5.05 (0.87) 

23 

4.95 (0.97) 

4.73 (0.94) 

4.63 (0.96) 

4.68 (0.96) 

24 

4.95 (0.97) 

5.35 (0.88) 

5.08 (1.04) 

5.10(1.02) 

Minimum 

4.21 

4.73 

4.22 

4.30 

Maximum 

5.60 

5.63 

5.58 

5.57 

Sample size 

14-21 

20-24 

103-133 

137-177 


Note. The importance scale is a 6-point scale: 1 ( not at all important ), 2 (of little importance), 3 (of some 
importance), 4 (moderately important), 5 (very important), 6 (extremely important)', respondents who judged the 
practice not relevant are not included in the calculation of the average importance judgment. The 24 CKT areas are 
summarized in the appendix. 

Importance judgments broken down by educator region are shown in Table 5. The 
differences in average importance judgments between regions (Northeast, Midwest, West, and 
South) differed most on CKT 8: Forms and Functions of Language; CKT 9: Text Craft and 
Structure; CKT 13: Presentation of Knowledge and Ideas; and CKT 17: Sources of Language 
Variability; differences on these areas ranged from 0.52 to 0.59 on a 6-point scale. Average 
importance judgments on all of these categories were higher for those from the Northeast 
compared to the Midwest with small to medium effect sizes ranging of 0.32 to 0.72. Average 
judgments for CKT 9 were also higher for those from the Northeast compared to those in the 
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South (ES = 0.43) and West (ES = 0.61) regions. Average judgments for CKT 13 were higher for 
those from the Northeast compared to those from the West (ES = 0.44), while average judgments 
for CKT 17 were lower for those from the Northeast compared to those from the South (ES = 
0.29). 


Table 5. Summary of Importance Judgments for Content Knowledge for Teachers (CKT) 
Areas by Geographic Region 


CKT area 

Northeast 

Midwest 

South 

West 

Overall 

1 

5.25 (0.91) 

5.19(1.06) 

5.31 (0.71) 

5.37 (0.75) 

5.28 (0.86) 

2 

5.47 (0.88) 

5.42 (0.84) 

5.58 (0.67) 

5.63 (0.54) 

5.53 (0.74) 

3 

5.43 (0.83) 

5.25 (0.90) 

5.58 (0.71) 

5.44 (0.78) 

5.43 (0.81) 

4 

5.54 (0.84) 

5.49 (0.80) 

5.68 (0.54) 

5.52 (0.63) 

5.57 (0.70) 

5 

5.30 (0.81) 

5.15 (0.88) 

5.37 (0.77) 

5.05 (0.79) 

5.23 (0.82) 

6 

5.06 (0.68) 

4.96 (0.88) 

5.27 (0.73) 

5.08 (0.83) 

5.11 (0.79) 

7 

5.32 (0.75) 

5.09 (0.88) 

5.40 (0.56) 

5.22 (0.88) 

5.26 (0.77) 

8 

4.88 (0.91) 

4.48 (0.79) 

5.03 (0.79) 

4.74 (0.89) 

4.79 (0.86) 

9 

5.21 (0.84) 

4.62 (0.80) 

4.83 (0.89) 

4.70 (0.81) 

4.81 (0.86) 

10 

4.92 (0.91) 

4.72 (0.90) 

4.89 (0.90) 

4.82 (0.79) 

4.83 (0.88) 

11 

5.09 (0.89) 

4.81 (0.95) 

5.10(0.78) 

5.00 (0.82) 

4.99 (0.86) 

12 

5.32 (0.71) 

5.23 (0.85) 

5.32 (0.62) 

5.20 (0.82) 

5.27 (0.74) 

13 

5.08 (0.77) 

4.56 (0.87) 

5.02 (0.75) 

4.70 (0.94) 

4.84 (0.85) 

14 

5.11 (1.04) 

5.06 (0.98) 

5.15 (0.79) 

5.14(0.78) 

5.12(0.89) 

15 

4.88 (0.95) 

4.46 (0.92) 

4.71 (0.89) 

4.79 (0.77) 

4.69 (0.89) 

16 

4.62 (0.89) 

4.71 (0.91) 

4.68 (0.88) 

4.83 (0.93) 

4.71 (0.90) 

17 

4.31 (0.88) 

4.02 (0.88) 

4.58 (0.99) 

4.24 (0.85) 

4.30 (0.93) 

18 

5.34 (0.91) 

5.00(1.01) 

5.13 (0.78) 

5.12(0.89) 

5.13 (0.90) 

19 

4.82 (0.83) 

4.41 (0.89) 

4.79 (0.91) 

4.71 (0.87) 

4.68 (0.89) 

20 

4.94 (0.92) 

4.72 (0.99) 

4.95 (0.82) 

5.03 (0.86) 

4.90 (0.90) 

21 

5.29 (0.83) 

4.82(1.07) 

5.07 (0.87) 

5.12(0.98) 

5.05 (0.95) 

22 

5.03 (0.90) 

4.91 (0.88) 

5.27 (0.71) 

4.93 (0.88) 

5.04 (0.84) 

23 

4.84 (0.93) 

4.69(1.01) 

4.78 (0.86) 

4.35 (0.95) 

4.67 (0.95) 

24 

4.97(1.01) 

5.00(1.03) 

5.23 (0.95) 

4.97(1.14) 

5.07(1.02) 

Minimum 

4.31 

4.02 

4.58 

4.24 

4.30 

Maximum 

5.54 

5.49 

5.68 

5.63 

5.57 

Sample size 

26-37 

41-54 

48-63 

34^12 

149-194 


Note. The importance scale is a 6-point scale: 1 ( not at all important ), 2 (of little importance), 3 (of some 
importance), 4 (moderately important), 5 (very important), 6 (extremely important); respondents who judged the 
practice not relevant are not included in the calculation of the average importance judgment. The 24 CKT areas are 
summarized in the appendix. 


Table 6 summarizes the most and least importance judgments for teachers and faculty 
members. Comparing relative importance judgments for teachers and faculty, the percentages 
differed by up to 16 percentage points; however, for most, the differences were less than 10 
percentage points. For least important, four CKT categories differed by more than 10 percentage 
points; for most important, four differed by more than 10 percentage points. 
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Table 6. Summary of Least and Most Important Judgments for Content Knowledge for 


Teachers (CKT) Areas 



Least important 

Most important 

CKT area 

Teachers 

Faculty 

Teachers 

Faculty 

1 

15.2% 

14.6% 

26.9% 

28.0% 

2 

10.2% 

12.2% 

28.4% 

26.8% 

3 

4.6% 

6.1% 

55.3% 

40.2% 

4 

4.1% 

9.8% 

56.9% 

43.9% 

5 

16.2% 

14.6% 

43.1% 

37.8% 

6 

16.2% 

18.3% 

17.8% 

15.9% 

7 

5.6% 

2.4% 

49.2% 

43.9% 

8 

34.5% 

37.8% 

5.6% 

3.7% 

9 

37.1% 

42.7% 

8.1% 

7.3% 

10 

39.6% 

40.2% 

4.1% 

6.1% 

11 

14.7% 

9.8% 

14.7% 

15.9% 

12 

8.6% 

11.0% 

27.9% 

19.5% 

13 

24.4% 

20.7% 

12.7% 

15.9% 

14 

21.3% 

17.1% 

14.7% 

14.6% 

15 

19.3% 

8.5% 

8.1% 

19.5% 

16 

39.6% 

51.2% 

3.0% 

8.5% 

17 

71.1% 

57.3% 

1.0% 

3.7% 

18 

15.2% 

14.6% 

25.4% 

23.2% 

19 

30.5% 

28.0% 

8.1% 

13.4% 

20 

17.3% 

14.6% 

19.8% 

20.7% 

21 

2.5% 

3.7% 

31.0% 

29.3% 

22 

8.1% 

9.8% 

11.7% 

14.6% 

23 

25.4% 

35.4% 

5.6% 

11.0% 

24 

18.8% 

19.5% 

20.8% 

36.6% 

Minimum 

2.5% 

2.4% 

1.0% 

3.7% 

Maximum 

71.1% 

57.3% 

56.9% 

43.9% 


Note. The 24 CKT areas are summarized in the appendix. 


To more easily digest the relative importance judgments, the judgments for least and 
most were combined to identify the top 10 CKT categories perceived to be most important for 
beginning elementary school teachers. First, the CKT categories were ranked by the percentage 
of teachers who identified the CKT areas as one of the five least important; the highest 
percentage received a rank of 24 and the lowest a rank of 1. Second, the CKT categories were 
ranked by the percentage of teachers who identified the CKT areas as one of the five most 
important; the highest percentage received a rank of 1 and the lowest a rank of 24. Then the two 
rankings were summed with the lower value indicating the more relative importance. 

Eight of the top 10 ELA CKT categories 4 were common between teachers and faculty. 
However, two CKT categories (CKT 11: Discussion and Collaboration and CKT 18: Programs, 
Routines, and Method for Language Arts Instruction) were flagged by teachers among their top 
10 but not by faculty. Faculty identified CKT 15: Integration and Application of Knowledge and 
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CKT 24: Development of Early Oral Language as being in the top 10, whereas teachers did not. 
While the orders varied, the top four CKT categories were the same for the two groups. 

While all 24 CKT categories were judged to be important (average judgment 4.2 or 
higher on a 6-point scale) by both teachers and faculty, the relative importance points out some 
differences between the two groups of educators, which are summarized in Table 7. 


Table 7. Top English Language Arts (ELA) Content Knowledge for Teachers (CKT) Areas 


and Practices by Relative Importance 


ELA CKT area 

Teachers 

Faculty 

1 : Print Concepts 

8 

7 (T) 

2: Alphabetic Principle 

5 (T) 

7(T) 

3: Phonological Awareness 

2 

2(T) 

4: Phonics and Word Recognition 

1 

2(T) 

5: Fluency 

7 

5 

7: Vocabulary 

4 

1 

11: Discussion and Collaboration 

10 (T) 

9 

12: Key Ideas and Details 

5(T) 

10 (T) 

15: Integration and Application of Knowledge 

18: Programs, Routines, and Methods for 

Language Arts Instruction 

16 

9 

6 

10 (T) 

21: Basic Processes of Reading and Writing 

3 

4 

22: Development of Word Reading 

10 (T) 

13 (T) 


Note. The 24 CKT areas are summarized in the appendix. (T) indicates a tie in the ranking of the relative 
importance. 


Discussion 

This study was designed to verify via a survey of a sample of educators that the ELA 
CKT measured by the ELA CKT component of the NOTE assessment series is relevant and 
important for the effective practice of beginning elementary school teachers teaching ELA. An 
online survey of educators — practicing elementary school teachers and college faculty who 
prepare teachers — judged the relevance and importance of 24 ELA CKT categories for beginning 
teachers. The content-related validity questions were couched in teaching ELA across the 
elementary school grade span. Across both groups of educators, each of the 24 CKT categories 
were judged to be relevant and important for beginning elementary school teachers, providing 
content validity evidence crucial for licensure examinations (cf. Ebel & Frisbie, 1991). 

This study design is not without limitations. Although our sample consisted of educators 
from around the country, our sample was neither nationally representative of beginning teachers, 
practicing teachers, or teacher education faculty. Generalizations from the sample provided to 
educators in general or to subgroups of interest need to be made with caution. We also recognize 
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that the ELA content framework includes a relatively large number of content dimensions and 
that the content defined under each dimension is itself extensive and complex. The data collected 
on agreement would be strengthened by evidence that participants understand the content 
dimensions as intended. 

When examining the relative importance of CKT, differences between elementary school 
teachers and college faculty who prepare elementary school teachers became apparent. Teachers 
identified CKT 11: Discussion and Collaboration and CKT 18: Programs, Routines, and Methods 
for Language Arts Instruction as two of the top 10 CKT categories; however, these two areas 
were not identified by faculty. In contrast, faculty identified CKT 15: Integration and 
Application of Knowledge and CKT 24: Development of Early Oral Language as two of their 
top 10, but teachers did not. Viewed separately, teachers and faculty viewed these four areas as 
important — at or above an average judgment of 4.0, or important, on a 6-point judgment scale, 
bindings such as these are consistent with prior research that has shown individuals in different 
positions may differentially weight the importance of varying aspects of performance for a given 
job (Motowidlo & Peterson, 2008). 

However, despite differences in ratings across teachers and faculty members, the 
relatively high level of agreement in deeming all 24 measured CKT categories relevant and 
important suggests this subset of content knowledge indeed represents information requisite for 
effective beginning practice. Overall, the content-related validity evidence collected supports the 
complete set of ELA content knowledge areas as important for a beginning elementary school 
teacher’s ability to be an effective educator. Each CKT was judged to be relevant and important, 
with 12 of the 24 areas receiving an average judgment of 5.0 or higher ( very important on the 6- 
point scale) by both teachers and faculty. 

Although sample sizes by ethnicity were relatively small, there were some differences 
across ethnicities in average importance judgments. Nonetheless, all three groups of respondents 
(White, Black, and Hispanic) indicated that each CKT area was at least moderately important. 
Similarly, while differences in average importance judgments between regions were observed, 
all regions reported that each content knowledge area was at least moderately important. 

Conclusion 

The agreement across the relevant subgroups of experts surveyed in this investigation 
suggests that all 24 CKT categories examined in this study are at least moderately important for 


ETS RM- 16-08 


18 



M. Martin-Raugh et al. 


Investigating the Relevance and Importance of ELA Content Knowledge Areas 


performing the job, suggesting each may be appropriate subject matter for a licensure 
assessment. Our results suggest that relevant subgroups of experts agree that having knowledge 
about these 24 CKT areas is reasonable for the effective practice of beginning elementary school 
teachers teaching ELA. 

The first step in developing an assessment is defining the construct being measured 
(Sireci & Sukin, 2013). In this study, we have collected judgments from educators about the 
relevance and importance of CKT thought to be central to effective practice for ELA teachers 
beginning their careers. This source of content-based validity evidence may be used to develop 
test specifications characterizing and operationally defining the job domain (Raymond, 2001; 
Sireci & Sukin, 2013). Although this investigation is an important step in the assessment 
validation process, future research is warranted to explore the extent to which test items map 
back to their intended content, identified by the results of the current survey. In this regard, 
alignment studies should be conducted to obtain this necessary evidence for supporting the use of 
the NOTE assessment series for teacher licensure. 
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Appendix. English Language Arts (ELA) Content Knowledge for Teaching (CKT) Areas 

1. Print Concepts 

This topic area requires students to demonstrate knowledge about print, such as knowing that 
written words represent spoken language, words are separated by spaces, text is written in a 
particular direction, and that there are distinguishing features of a sentence (e.g., capitalization 
and punctuation). Students should also be able to differentiate between the pictures and printed 
words on a page. 

2. Alphabetic Principle 

This topic area requires students to demonstrate knowledge that print is a representation of sound 
and that words are made up of both individual sounds as well as patterns of groups of sounds. 
This awareness consists of knowledge of the alphabet’s uppercase and lowercase letter names, 
letter shapes, and corresponding sounds. Students should understand that the individual 
phonemes (the smallest units of sound) they hear in words are represented by graphemes (the 
alphabetic letters) and that those sounds can be analyzed and synthesized in the decoding and 
encoding process. 

3. Phonological Awareness 

This topic area requires students to demonstrate implicit and explicit knowledge about the lexical 
and sublexical structure of oral language. Students understand that speech is composed of 
various phonological units (phonemes, morphemes, syllables, and words) that vary in size. 
Students have phonological awareness and can detect and manipulate speech sounds at several 
levels: word parts (e.g., the parts of compound words), syllables (e.g., “cow-boy”), onset and 
rime (e.g., for “ball,” the onset is the beginning sound Pol and the rime is the ending sound /all/), 
and phonemes (e.g., /b/, /a/). 

4. Phonics and Word Recognition 

This topic area requires students to decode unfamiliar words using grade-appropriate phonics and 
word- analysis skills. Students should pronounce unfamiliar words by systematically applying 
knowledge of letter-sound correspondence and orthographic patterns and by making word 
analogies (e.g., “bolt” sounds like “colt” but starts with /b/). Along with understanding basic one- 
to-one letter-sound correspondences, students accurately read multisyllabic words in and out of 
context by breaking words into syllables, identifying affixes (i.e., prefixes and suffixes), and 
using strategies such as word analogies. Students should also read grade- appropriate high- 
frequency words by sight. 

5. Fluency 

This topic area requires students to translate (or read) text orally and silently with accuracy and 
automaticity in the service of text comprehension. Students are able to read grade-level text with 
purpose and understanding as well as read appropriate texts with accuracy, at the appropriate 
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rate, and with prosody (i.e., resembling natural speech in stress, pitch, phrasing, and timing) on 
successive readings. Students should also use context to confirm or self-correct word recognition 
and understanding, rereading words and phrases when necessary. Fluency also includes 
demonstration of sufficient stamina to finish a reading task. 

6. Conventions of Standard Academic English 

This topic area requires students to demonstrate knowledge of the academic English that 
characterizes both oral discourse and texts of schooling (in addition to having competence in 
their first language and/or dialect). Thus, students must demonstrate command of academic 
English grammar, capitalization, punctuation, and spelling. Grammar content involves the 
structural rules that govern clauses, phrases, and words, including conventional use of word 
tense, parts of speech (e.g., nouns, verbs, adjectives), subject-verb agreement, and correlative 
conjunctions (e.g., “either/or,” “neither/nor”). Students should also have knowledge of how to 
produce simple, compound, and complex sentences. Students learn to spell grade-appropriate, 
irregularly spelled words by applying conventional knowledge of alphabetic spelling, common 
orthographic patterns, syllables and affixes, as well as derivational suffixes (“compete” versus 
“competition”). 

7. Vocabulary 

This topic area requires students to demonstrate knowledge of a depth and breadth of words 
through listening, speaking, reading, and writing. Students know the denotative meanings and 
uses of academic words, domain- specific vocabulary, and words central to understanding and 
writing about topics they are studying as well as connotative meanings represented through 
figurative and idiomatic language. However, students also take an active role in analyzing and 
working out the meanings of unfamiliar words or new uses of familiar words by using key 
strategies: clarifying unknown word meaning through context (e.g., through semantic and 
syntactic cues), using knowledge of word parts (e.g., morphological parts, such as affixes and 
roots), making word associations (e.g., antonyms, synonyms, cognates), and utilizing external 
resources (e.g., dictionaries and feedback from peers) to aid in pronunciation, meaning, and word 
usage in reading and writing. 

8. Forms and Functions of Fanguage 

This topic area requires students to demonstrate knowledge about how language and its 
conventions affect meaning; this knowledge supports comprehension (reading and listening) and 
making effective choices for meaning and style in speaking and writing. Students must discern 
the appropriate level of formal language use across various contexts as well as analyze the use of 
English dialect and register within and across texts. In their own speaking and writing, students 
must reach beyond conventional appropriateness and select words, phrases, and punctuation for 
effect and precision. Students also make choices in how to expand, reduce, and combine 
sentences in order to infuse meaning, interest, and style. 


ETS RM- 16-08 


24 



M. Martin-Raugh et al. 


Investigating the Relevance and Importance of ELA Content Knowledge Areas 


9. Text Craft and Structure 

This topic area requires students to demonstrate knowledge about the language of written texts as 
a matter of craft. It entails analysis of how printed language, such as specific word choice, is 
used to convey meaning and tone. This topic also includes analysis of how parts of a text (e.g., 
paragraphs, chapters, scenes) relate to one another or how text features help readers locate 
relevant information efficiently. It also includes cross-text analysis, for example, in narrative 
text, comparing how authors convey point of view differently for the same event or topic or, in 
informational text, comparing the overall structure (comparison, chronology, cause and effect) of 
an event or topic across multiple texts. 

10. Text Types 

This topic area requires students to demonstrate knowledge about different genre types — 
conventional macrostructures for organizing texts that are related to unique purposes (e.g., 
narrative, descriptive, persuasion, exposition, and directions). It also includes knowledge of 
typical elements of different genres (e.g., narration, dialogue, description, quotations, concrete 
facts and details, and examples). Across all text types, it includes use of microstructures such as 
transitional words, phrases, and clauses to link ideas (e.g., “first,” “next,” “then,” 

“consequently,” “specifically”) and rhetorical predicates (i.e., midlevel organizational structures 
or frames, such as problem and solution, conflict and resolution, label and list). Additionally, it 
includes using formats for introducing, sequencing, and concluding all types of texts. Within 
narratives, it includes communicating real or imagined experiences or events using effective 
techniques, such as providing descriptive details and concrete, sensory detail and sequencing 
events through a narrator, a dialogue, or a description. Within opinion pieces and informative or 
expository text, the topic area includes developing a clear introduction topic with supporting 
facts and providing concrete details that are logically grouped and organized. 

11. Discussion and Collaboration 

This topic area requires students to prepare for and participate in a range of conversations and 
collaborations with diverse peers. Participating in collaborative discussion includes using social 
knowledge of discourse conventions such as how to enter and hold a conversation (e.g., taking 
turns, acknowledging others’ comments, clarifying information, and building on others’ ideas), 
being considerate and respectful of others, as well as knowing how to use the conventions to 
communicate clearly and persuasively. Beyond abiding by the conventions for conversing, 
students utilize group discussions to build knowledge and comprehension. They are able to 
paraphrase and summarize a text or the speaker’s main points, reasons, and evidence. They are 
also able to express their own ideas and feelings and build on the ideas of others clearly and 
persuasively. Additionally, they integrate and evaluate information by posing and responding to 
discussion questions and by explaining how evidence, reasoning, and point of view are 
connected to another’s claim. They regulate their interpretation of texts or sources of information 
by reflecting on and evaluating others’ perspectives. 


ETS RM- 16-08 


25 



M. Martin-Raugh et al. 


Investigating the Relevance and Importance of ELA Content Knowledge Areas 


12. Key Ideas and Details 

This topic area requires students to read closely to determine what the text explicitly says and to 
make logical inferences and cite specific textual evidence in support of conclusions. For 
example, students should be able to ask and answer questions to determine understanding of a 
text and refer to the text as support for answers. It also includes determining central ideas or 
themes and summarizing or paraphrasing the key supporting details, evidence, and ideas. 
Students should be able to recount stories to determine the central message, lesson, or moral and 
explain how those are conveyed through details. This topic also includes establishing 
relationships within a text between characters or individuals, settings, events, ideas, or concepts 
based on specific text information (for example, determining a connection between a theme and 
a series of events or noting how different characters respond to challenges). 

13. Presentation of Knowledge and Ideas 

This topic area requires students to present information in an organized and stylistic manner 
appropriate to the audience and purpose. For example, students should sequence ideas logically, 
use appropriate facts and relevant descriptive details to support main ideas, establish a line of 
reasoning and organization, and speak clearly at an understandable pace. This topic also includes 
adapting the level of formal speech appropriate to a variety of contexts and tasks, such as 
speaking in complete sentences or selecting the appropriate dialect or register. Additionally, this 
topic encompasses students’ strategic use of digital and visual media displays to enhance 
expression and comprehensibility of ideas. 

14. Production of Written Texts 

This topic area requires students to effectively produce and distribute writing. It includes the 
production of clear and coherent writing by adapting the organization and style of written 
information to the audience, task, and purpose. It also includes taking a piece of written work 
through the stages of the writing process (e.g., planning, drafting, revising, editing, and 
rewriting) as well as producing first draft, on-demand writing. Students should demonstrate the 
ability to sustain fluent writing long enough to produce required products. Part of effective 
writing production includes the ability to produce and publish texts using a keyboard or digital 
device. 

15. Integration and Application of Knowledge 

This topic area requires students to integrate and evaluate information and ideas across various 
individuals, formats, and media. Students should be able to understand and critique the validity 
of arguments, evaluate the validity of reasoning, determine the relevance and sufficiency of 
evidence, and identify the relationship between evidence and reasoning and a claim. Students 
should also be able to integrate information across multiple texts in order to synthesize the 
information quickly and efficiently, compare different author approaches or ideas, or analyze 
how various formats contribute to the meaning, tone, or beauty of text. Students should be able 
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to apply information and ideas to new contexts and problems and integrate information in order 
to write or speak about a subject knowledgeably. 

16. Research to Build and Present Knowledge 

This topic area requires students to conduct research in order to gather relevant information 
around a subject under investigation. This includes locating, selecting, gathering, recalling, 
categorizing, and reorganizing relevant information from different text types to support analysis. 
The topic also includes analyzing and reflecting on evidence by comparing and contrasting 
characters, settings, and events (for narrative texts), explaining how an author uses reasons and 
evidence to support particular points (for informational texts), as well as identifying 
corresponding reasons and evidence. It also includes the ability to determine the credibility, 
accuracy, and bias of sources. 

17. Sources of Language Variability 

This topic area includes knowledge about why and how language variations influence literacy 
development and participation. It requires knowing the many factors that contribute to language 
variability among individuals, such as having expertise in both a home language (if different 
from English) and a second language (i.e., English), the use of a dialect, and/or experience with 
the registers of oral and written academic English used in school settings. It is also includes 
understanding the linguistic implications (e.g., pronunciation) and the cultural implications (e.g., 
cultural norms) of language variability and how it affects student performance across literacy 
activities. It also includes having knowledge about the instructional practices that best utilize 
home language competence to further develop school language competence. 

18. Programs, Routines, and Methods for Language Arts Instruction 

This topic area includes knowledge of general approaches for teaching literacy (e.g., basal and 
core reading programs, language experience, balanced literacy), routines and activity structures 
for teaching reading (e.g., guided reading, book clubs, read-alouds), writing (e.g., writing 
workshop, genre study), oral language (e.g., Socratic seminars, discourse routines, literature 
circles), and other widely used methods (e.g., use of mentor texts, fluency-building activities, 
comprehension-strategy instruction, vocabulary instruction, word-recognition activities, phonics 
instruction). 

19. Interaction of Knowledge, Language Processes, and Learning 

This topic area includes knowledge about the essential role that content knowledge plays in 
comprehending and composing texts. It also includes knowledge about the synergistic 
relationship between content knowledge and language comprehension. World and topical 
knowledge facilitate both oral and written language comprehension, which, in turn, enables 
learning, leads to changes in content knowledge, and thereby supports subsequent cycles of 
comprehension, language development, and knowledge growth. This topic also includes 
knowledge about direct and indirect instructional approaches for developing students’ content 
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knowledge, making explicit connections between new information and students’ background 
knowledge, and promoting additional student learning by encouraging wide reading. 

20. Role of Engagement in Reading and Writing 

This topic area includes knowledge about the fundamental intrapersonal and interpersonal factors 
that influence cognition and learning. Included are factors of motivation (both intrinsic and 
extrinsic), interest (both personal and situational), metacognitive processes (e.g., self-assessment, 
comprehension monitoring), and conscientiousness (i.e., the capacity to maintain stamina and 
persistence in the face of challenge and difficulty), and literate identity (i.e., a sense of self- 
efficacy in language activities). This topic area also includes knowledge of how to facilitate 
motivation, engagement, efficacy, and stamina through choice (of books to read, topics for 
writing, and artifacts to represent one’s learning), encouragement (to promote persistence, 
situational interest, and literate identity), feedback (to enhance self-assessment and self-efficacy), 
collaboration (to enhance literate and community identities), and curricular coherence (to 
promote literate identity and self-efficacy). 

21. Basic Processes of Reading and Writing 

This topic area includes knowledge about skilled reading and writing performance. In reading, it 
involves the understanding of how processing at various levels of orthography (letters, letter 
clusters, syllables, morphemes, and words), language (phonological, phonemic, morphemic, 
lexical, semantic, syntactic, rhetorical, and pragmatic), and knowledge stored in long-term 
memory results in accurate decoding, fluent oral and silent reading, and text understanding. In 
writing, it involves all the same levels of processing knowledge, language, and orthography, but 
in the reverse order — from knowledge to language to orthographic representation. This topic also 
includes the understanding that these factors interact in complex and nonlinear ways in both 
comprehension and composition. 

22. Development of Word Reading 

This topic area includes knowledge about the typical ways in which reading and writing develop 
along with an understanding that the stages in typical development must be regarded as flexible 
heuristics rather than rigid, sequential phases. The stages may vary from theory to theory, but 
they usually include a pre- alphabetic stage (students attend to idiosyncratic aspects of word form, 
such as the long tail in “monkey”), an alphabetic stage that is often divided into partial (attending 
to salient or initial letters) and full (attending to all the letter information), and a consolidated 
stage (balancing orthographic, phonological, contextual, and conceptual information to achieve 
fluent decoding on the way to comprehension). It also includes the knowledge that fluent readers 
have many ways to read words accurately: as sight words (little attention is paid to the 
component orthographic features), sequential decoding (students decode each letter and blend the 
sounds together into a word), analogical decoding (the word is recognized as a member of a 
family of familiar endings: “goad,” “road,” “toad”), and employing context clues with letter- 
sound knowledge (e.g., “I know it means ‘to spur on,’ and it ends like ‘road,’ so it must be 
‘goad’”). 
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23. Development of Word Spelling 

This topic area includes knowledge about the general course of spelling development from 
precommunicative (such as when scribbling and drawing reveal a sensitivity to representation 
but not necessarily communication) through a partial phonetic stage (in which marks made on 
paper reveal an attempt to represent sounds) through a phonetic stage (where both errors and 
accurate representations reveal an attempt to represent sounds) through a transitional stage (in 
which patterns of letters are predominant in children’s representations of sounds) and into an 
integrated stage (where sensitivity to syllabic and morphemic-base words and affixes are salient). 
Also included is the understanding that progression through these stages is flexible and variable 
across students and the sequence is governed by children’s perceptions of patterns of regularity 
in sound-symbol mappings at every stage. 

24. Development of Early Oral Language 

This topic includes knowledge about the foundational role that oral language development plays 
in virtually every aspect of written language development. Early oral language performance both 
predicts and explains later written language performance. The implication for preschool and 
primary grade instruction is that teachers must provide substantial opportunities for students to 
engage in word work (e.g., rhyming, alliteration, and elision activities) and substantive talk about 
ideas, experiences, and activities that students encounter in home and school life. 
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Notes 

1 The advisory panel included the following members: P. David Pearson, advisory chair, UC 

Berkeley; Devon Brenner, Mississippi State; Joanne Carlisle, University of Michigan; Carol 
Connor, Arizona State; Vicki Benson Griffo, UC Berkeley; Virginia Goatley, University at 
Albany; James Hoffman, UT Austin; Barbara Kapinus, NEA, Smarter Balance; Jeannette 
Mancilla-Martinez, UC Irvine; and Julie Washington, Georgia State University. In addition, 
assessment experts and content research staff from ETS provided input and contributed to the 
development of the framework: Gary Sykes, Andrew Croft, Sally Gillespie; David Kirui, and 
Geoffrey Phelps (Understanding Teaching Quality Center); Allison Brettschneider and Eric 
Steinhauer (Assessment Development). 

2 Response rate for the emailed surveys delivered to sampled educators. 

1 Tannenbaum and Rosenfeld (1994) recommended that an average importance judgment of 3.5 
on a 5-point scale was sufficient to determine importance for licensure. Translating this 
finding to a 6-point scale would result in a threshold of 4.2. 

4 Due to ties in the rankings, 1 1 CKT categories were identified for teachers and for faculty. 
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